Pipleline merge operations using source data and multiple destination data structures

ABSTRACT

A source data structure may be scanned one time in order to obtain source data that is then used for multiple merge operations in order to merge source data into one or more destination data structures. Each merge operation is then performed using the same scan of the source data structure and a scan of the destination data structure.

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application is related to U.S. patent application Ser. No.______ (Attorney Docket No, 50277-2098), entitled METHOD AND APPARATUSFOR PERFORMING MULTI-TABLE MERGE OPERATIONS IN A DATABASE ENVIRONMENT,by RICHARD YU GU, HARMEEK SINGH BEDI and ASHISH THUSOO, filed on thesame date as this application, the content of which is herebyincorporated by reference in its entirety.

[0002] This application is related to U.S. patent application Ser. No.______ (Attorney Docket No, 50277-2099), entitled METHOD AND APPARATUSFOR PERFORMING MULTIPLE MERGE OPERATIONS USING SOURCE DATA THAT ISMODIFIED IN BETWEEN THE MERGE OPERATIONS, by RICHARD YU GU, HARMEEKSINGH BEDI and ASHISH THUSOO, filed on the same date as thisapplication, the content of which is hereby incorporated by reference inits entirety.

[0003] This application is related to U.S. patent application Ser. No.______ (Attorney Docket No, 50277-2101), entitled TECHNIQUE FOR USING ACURRENT LOOKUP FOR PERFORMING MULTIPLE MERGE OPERATIONS USING SOURCEDATA THAT IS MODIFIED IN BETWEEN THE MERGE OPERATIONS, by RICHARD YU GU,HARMEEK SINGH BEDI and ASHISH THUSOO, filed on the same date as thisapplication, the content of which is hereby incorporated by reference inits entirety.

FIELD OF THE INVENTION

[0004] The present invention relates to database operations andmanagement. In particular, the invention relates to method and apparatusfor performing pipeline merge operations between a source data structureand a plurality of destination data structures.

BACKGROUND OF THE INVENTION

[0005] In a data warehouse environment, tables need to be refreshedperiodically with new data arriving from client systems. The new datamay contain changes to existing records, i.e., rows in tables, of thedatabase and/or new records that need to be inserted.

[0006] A data manipulation operation is defined as an operation, whichmodifies a data set. Examples of data manipulation operations inStructured Query Language (SQL) include UPDATE, INSERT, DELETE, andMERGE. In the context of our invention, we consider those forms of datamanipulation operations where a source data set is compared with adestination data set in order to generate modifications to the latter.This can be achieved today through UPDATE, INSERT, DELETE, and MERGEstatements. All these statements modify a single target data set. Suchstatements have been used with, for example, the Oracle 9i databasesystem.

[0007] Another feature, provided by the SQL statement MERGE, combines aconditional INSERT, UPDATE and DELETE commands in a single atomicstatement to merge data from a source to a destination. The INSERT,UPDATE, DELETE commands in the context of MERGE command are consideredconditional in that (a) if a record in the new data corresponds to anitem that already exists in the destination, then an UPDATE and possiblyDELETE operations are performed on the item; and (b) if a record in thenew data does not already exist in the destination, then an INSERToperation is performed to add a corresponding record to the destination.

[0008] Database application such as data warehouses often require datafrom a source structure to be merged into multiple destinationstructures. FIG. 10 illustrates a typical plan for a database systemthat merges data from a source table 1010 into multiple destinationtables within the database system. The multiple destination tables areillustrated by a first destination table 1020 and a second destinationtable 1025. To perform the MERGE operations, a first source scan 1012 isperformed on the source table 1010, and a first destination scan 1022 isperformed on the first destination table 1020. The first source scan1012 and first destination scan 1022 may be completed at time T0. Oncethe scans are performed, a first MERGE operation 1030 is performed tomerge data from the source table 1010 into the first destination table1020. The first MERGE operation 1030 determines, for each row beingmerged into the destination table, whether the row corresponds to a rowthat is already in the destination table.

[0009] To perform the second MERGE operation 1040, a second source scan1014 is performed on source table 1010. A second destination scan 1024is also performed on second destination table 1025. The second sourcescan 1014 and the second destination scan 1024 are completed at time T1.Once the scans are completed, the second MERGE operation 1040 isperformed.

[0010] The plan of FIG. 10 illustrates the manner in which successiveMERGE operations between a source data structure and other destinationdata structures are typically performed. Each MERGE operation requires ascan of the source data structure. This can be problematic when thesource data structure is large, or otherwise be sufficiently complex torequire an expensive and lengthy process to be scanned. As a result,when the source data structure is subjected to multiple MERGEoperations, the individual MERGE operation can consume significantcomputational resources for a lengthy period of time.

BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present invention is illustrated by way of example, and notby way of limitation, in the figures of the accompanying drawings and inwhich like reference numerals refer to similar elements and in which:

[0012]FIG. 1 is a block diagram of a database system configuredaccording to an embodiment of the invention.

[0013]FIG. 2 illustrates a plan for performing a multi-table MERGEoperation, under an embodiment of the invention.

[0014]FIG. 3 is a block diagram illustrating a system where a sourcedata stream is subjected to multiple MERGE operations to merge differentportions of the source data stream with different destination tables.

[0015]FIG. 4 illustrates a plan for performing a multi-table merge wherea source data stream used in performing multiple MERGE operations ismodified by one of the MERGE operations before another of the MERGEoperations is performed.

[0016]FIG. 5 illustrates implementation of an embodiment in astar-schema.

[0017]FIG. 6 illustrates a plan for providing a pipeline for enabling asource data stream to be concurrently merged into multiple destinationdata structures.

[0018]FIG. 7 is a plan that illustrates use of a lookup node to enableaugmenting source data in between MERGE operations when the source datais to be used for consecutive MERGE operations, under an embodiment ofthe invention.

[0019]FIG. 8 illustrates a method for using a lookup node to augmentsource data as a result of performing a MERGE operation for use with asubsequent MERGE operation.

[0020]FIG. 9 is a block diagram illustrating hardware of a computersystem for use with an embodiment of the invention.

[0021]FIG. 10 is a prior art plan that illustrates data from a sourcetable being merged with multiple destination tables.

DETAILED DESCRIPTION OF THE INVENTION

[0022] A method and apparatus for performing multi-table mergeoperations are described. In the following description, for the purposesof explanation, numerous specific details are set forth in order toprovide a thorough understanding of the present invention. It will beapparent, however, that the invention may be practiced without thesespecific details. In other instances, well-known structures and devicesare shown in block diagram form in order to avoid unnecessarilyobscuring the invention.

General Overview

[0023] A merge operation is a data manipulation operation that refers toa process where two sets of data are compared and possibly combined. Ifthe result of the comparison is that the two sets of data areequivalent, then the result of the data manipulation operation may bethat neither set of data is modified. If a difference is determined,then the result of the operation may be that one set of data is modifiedbased on the other set of data. In the context of database operations,each merge includes identifying differences between data from the sourcedata stream and data from one of the destination data structures, andthen modifying that destination data structure based on the identifieddifferences, if any. Examples of merge operations (or other datamanipulation operations) in SQL include UPDATE, INSERT, DELETE andMERGE. Throughout much of this application, the specific datamanipulation operation discussed is MERGE.

[0024] Embodiments described herein provide for performing multiplemerge operations to integrate data from a source data structure with oneor more destination data structures. Only one scan of the source datastructure is necessary to obtain the source data for performing all ofthe merge operations. Embodiments such as described herein conservesubstantial processing resources and time by enabling multiple-mergeoperations to be performed using only a single scan of the source datastructure.

[0025] According to one embodiment, data from a source data structure iscombined with multiple destination data structures using a single scanof the source data structure, where the data structures involved in themerge operation are relational data structures. In an embodiment, aplurality of merge operations are performed to combine a source datastream obtained from the source data structures with data from one ormore of the destination data structures, but the same scan of the sourcedata structure is used to obtain the source data stream that is thebasis for performing all of the merge operations.

[0026] According to another method, a merge operation is performed tomerge a source data stream into a first destination data structure. Themerge operation augments changes to the source data stream for use insubsequent merge operation. In one embodiment, the source data stream isstored with its changes in one or more intermediate data structureduring performance of the merge operations. A determination is made atthe intermediate data structure to determine how the source data is tobe modified for subsequent merge operations.

[0027] In another embodiment, the source data stream, which may havebeen modified by the first merge operation, is pipelined to thesubsequent merge operations. As a result of this, multiple mergeoperations are active on different portions of the source stream at thesame time.

[0028] According to another embodiment, a plurality of merge operationsis performed using a single scan of a source data structure. Each of themerge operations is an operation to merge or otherwise combine a sourcedata stream into at least one of a plurality of destination datastructures. The performance of at least one of the plurality of mergeoperations causes data from the source data stream to be augmented forsubsequent merge operations.

System Description

[0029]FIG. 1 illustrates a database system configured according to anembodiment. A database system 100 such as shown by FIG. 1 may correspondto systems, which communicate with numerous external data sources tocombine data into a centralized source. An example of such a system isORACLE WAREHOUSE BUILDER, manufactured by ORACLE CORP.

[0030] In an embodiment, database system 100 includes a databasemanagement component (DMC) 130. The DMC 130 illustrates components andresources of the database system 100 which are used to receive data fromexternal sources and to merge external data into internal datastructures of the database system. In an embodiment, the internal datastructures managed by the DMC 130 are in the form of tables. In anexample provided by FIG. 1, the destination data structures include afirst destination table 120, a second destination table 122, and a thirddestination table 124. In one embodiment, data may be imported into thedatabase system 100 from an external data source 105. The external datasource 105 may correspond to another database system, computer system,storage device, or computer-readable memory that can provide data todatabase system 100.

[0031] In FIG. 1, a set of source data 110 is received from the externaldata source 105. The source data 110 may correlate to data copied from asource table 108 (or other relational data structure) residing withinthe external data source 105. The DMC 130 merges source data 110 intodestination tables 120, 122, and 124. The DMC 130 merges the source data110 by performing a series of MERGE operations to combine the sourcedata with each of the destination tables 120, 122 and 124. In oneembodiment, each MERGE operation between the source data 110 and one ofthe destination tables 120, 122, and 124 results in data being updatedor inserted in one or both of the source data 110 and the correspondingdestination table. To perform the MERGE operations, the DMC 130 scanseach of the destination tables 120, 122 and 124, and the source table108. The scan of the source table 108 results in the source data 110,which is then used for the subsequent MERGE operations. The source data110 may be in the form of a stream. As will be described with someembodiments of the invention, the source data 110 may mutate orotherwise be modified in between subsequent MERGE operations.

[0032] According to an embodiment, the DMC performs a single scan of thesource table 108 in order to merge data from the source table into eachof the destination tables 120, 122 and 124. A scan 112 of the sourcetable 110 may be performed to obtain the source data 110 prior to any ofthe MERGE operations being executed. A first destination scan 142 of thefirst destination table 120 is performed to merge some or all of thesource data 110 into the first destination table 120. A seconddestination scan 144 of the second destination table 122 is performed inorder to perform a second MERGE operation where the source data ismerged into the second destination table 122. In performing the secondMERGE operation, another scan of the source table 110 is not performed.A third MERGE operation may be performed in order to combine the sourcedata 110 with the third destination table 124. In performing the thirdMERGE operation, another scan of the source table 108 is not performed.In this way, a multi-table merge is performed using only the single scan112 of the source table 108 that yielded the source data 110. The totalnumber of scans used to perform the multi-table merge is n+1, wherein nis the number of destination tables being merged with the source table110.

[0033] While FIG. 1 illustrates use of a multi-table MERGE operationwith tables as source and destinations, other embodiments may use otherforms of data structures. For example, in one embodiment, the sourcetable 108 may be a relational data structure such as rows of data thatare the result of a query to another table or relational data structure.Thus, the source data 110 may be in the form of a stream of query resultfrom some relational data structure.

Multi-Table Merge

[0034]FIG. 2 illustrates a plan for performing a multi-table merge,under an embodiment of the invention. In FIG. 2, source data 210 iscombined with a data stream from a first destination table 220, and thenwith a second destination table 225. By time T0, a source scan 212 isperformed to yield source data 210, and a first destination scan 222 offirst destination table 220 is completed.

[0035] Once the source scan 212 and the first destination scan 222 arecompleted, a first MERGE operation 230 is performed. The first MERGEoperation 230 includes operations that identify differences betweensource data 210 and first destination table 220. The first destinationtable 220 may be modified to account for the changes with the sourcedata 210. In an embodiment such as described with FIG. 4, the sourcedata stream generated from scan 210 can be augmented by the MERGEoperation.

[0036] In an embodiment, a MERGE command may comprise two conditionalcommands: UPDATE and INSERT. In UPDATE, data in first destination table220 is modified according to corresponding elements of source data 210.In INSERT, data from source data 210 is augmented and/or inserted tofirst destination table 220.

[0037] After first MERGE operation 230 is completed, a seconddestination scan 224 of second destination table 225 is completed attime T1. Another scan of source data 210 is not performed. Rather,second MERGE operation 240 is performed using the source scan 212 andthe second destination scan 224. The second MERGE operation 240 mayperform functions similar to the first MERGE operation 230.

[0038] Thus, an embodiment such as described with FIG. 2 preserves thesource data 210 that results from source scan 212 for the subsequentMERGE operations. The source data stream from source scan 212 may bepreserved by performing an “outer-join” operation before actuallyperforming the MERGE operation. An outer-join operation is a type ofjoin operation, where overlap between the source data 210 and thedestination data structure is identified, except that the outer-joinoperation also preserves the source data.

[0039] Syntax for accomplishing one type of MERGE operation (the MERGEcommand) such as detailed in FIG. 2 is provided by a first set ofinstructions, illustrated below. This MERGE operation includes anouter-join operation: 10* merge 20* using <source> 30*  into<destination> 40*  on <predicate> 50*  when matched then 60*   update70*   set <destination column> 80*  when not matched then 90*   insert<columns >

[0040] The syntax example provided above defines the source data anddestination structures in lines 20 and 50. The “into” clause of line 30causes the first destination table scan 222 to be performed. The “using”clause of line 20 causes the source table scan 212 to be performed. The“on” clause in line 40 defines the condition by which a comparison ismade between the source data 210 and first destination table 220. Forexample, the “on” clause may specify that one or more column of firstdestination table 220 are to be matched to specified columns of sourcedata 210. If the predicate of the “on” clause is true, then an update isperformed in line 60 and 70 on the first destination table 220. Theupdate consists of setting the destination columns in line 70 to aparticular set of values defined by the specified columns of source data210. If the predicate of the “on” clause is false, then an insert may beperformed to augment the first destination table 220 with values derivedfrom columns specified in the source data 210. The “on” clause may beimplemented as an “outer join” operation that happens before the MERGEoperation is performed. The outer-join is a join operation between thesource data stream and the destination data where the source data ispreserved, regardless of whether or not all of the rows of the sourcedata stream match with a corresponding row of the destination data. If asource row matches with a destination row, the result of the outer-joinoperation is the source columns and the destination columns of thejoined rows from the respective data streams. If a source row does notmatch with any destination row, then the result of the outer-joinoperation consists of the column values of the source row, and NULLcolumn values for the destination columns.

[0041] According to another embodiment, different portions of the sourcedata 210 may be merged with different destination tables in a series ofMERGE operations. FIG. 3 illustrates fan-out of the source data 210 intomultiple destination tables. The source data stream 310 may be subjectedto multiple MERGE operations in order to merge different portions of thesource data with a plurality of destination tables. In FIG. 3, theplurality of destination tables is provided by a first destination table320, a second destination table 322, and a third destination table 324.

[0042] As described previously with FIG. 2, a plurality of MERGEoperations may be performed using a single scan of source table. In anembodiment such as described in FIG. 3, different portions of the sourcedata stream 310 may be subjected to a MERGE operation with a differentone of the destination tables 320, 322, 324 respectively. The particularportion of the source data stream 310 that is merged with eachdestination table 320, 322, 324 may be dependent on the predicatespecified for the MERGE operation between the source data stream 310 andthe specified destination table.

[0043] For example, as shown in FIG. 3, a first section 312 (i.e. row 1,row 2) of source data stream 310 matching a first predicate 332 ismerged with first destination table 320. A second source section 314(i.e. row 1, row 2, row 3) of source data stream 310 matching a secondpredicate 334 is merged with second destination table 322. A thirdsource section 316 (i.e. all of source data stream 310) matching a thirdpredicate condition 336 is merged with third destination table 324. Inthis way, the multi-table merge may be performed using a single scan ofthe source data structure that yielded source data stream 310, exceptthat different portions of that source data are merged with differentdestination tables 320, 322, 324. The different portions (which may ormay not overlap) of the source data 210 may be identified from a streamor intermediate data structure that results from the scan of the sourcedata structure.

[0044] In an embodiment, each predicate condition may be an “all”condition or a “first” condition. When an “all” is used, the result isthat that each designated unit (i.e. row) of source data stream 310 ismatched with all of the predicates and is used by the MERGE operation ofthe matching predicates to merge that unit of the source data with thecorresponding one of the destination tables 320, 322, 324. When a“first” condition is used, the result is that each designated unit (i.e.row) of source data stream 310 is used by the MERGE operationcorresponding to the “first” predicate that it matches with and isdisregarded by all subsequent predicates. The order in which thepredicates are evaluated is the order in which the predicates appear inthe statement. Therefore in the second set of instructions shown, if“all” is specified, then each source row is matched with each of thepredicates at line 15, 55 and 95 and the corresponding merge isperformed. On the other hand if “first” is specified a row is matchedwith predicate at line 15, if it matches, the corresponding merge isexecuted and the processing of the row ends, if it does not match, therow in matched with predicate in line 55 and so on.

[0045] A suitable syntax for performing an embodiment such as describedin FIG. 3 is provided by a second set of instructions, illustratedbelow. 10* merge <all/first> using <source> 15* when <predicate 1> 20* into <destination 1> 25*  on <sub-predicate 1> 30*   when matched then35*    update 40*    set <columns > 45*   when not matched then 50*   insert <columns > 55* when <predicate 2> 60*  into <destination 2>65*  on <sub-predicate 2> 70*   when matched then 75*    update 80*   set <columns > 85*   when not matched then 90*    insert <columns >95* when <predicate 3> 100*  into <destination 3> 105*  on<sub-predicate 3> 110*   when matched then 115*    update 120*    set<columns > 125*   when not matched then 130*    insert <columns >

[0046] The second set of instructions may be executed to implement anembodiment such as described in FIG. 3. For purpose of description, thesecond set of instructions will be described in the context of FIG. 3.The source data stream 310 results from scanning the source datastructure, provided by the command of “using <source>” in line 10. Theline 10 specifies whether the predicates 332, 334, 336 of themulti-MERGE operations are to be treated as type “all” or “first”predicates. The destination tables 320, 322, 324 are scanned by the“into” clauses in lines 20, 60, and 100. The MERGE operation performedfor merging the portion of the source data 310 matching first predicate332 is illustrated by lines 15-50. Likewise, the MERGE operationperformed for merging the portion of the source data stream 310 matchingsecond predicate 334 is illustrated by lines 55-90. The MERGE operationperformed for merging the portion of the source data stream 310 matchingthe third predicate 336 is illustrated by lines 95-130. Each of the “on”clauses provided in lines 25, 65, 105 are to initiate a comparisonbetween the matched source data stream 310 and each of the destinationtables 320, 322, 324. The second set of instructions ensures that the“on” clauses are performed so as to preserve the scan of the source datastream 310. This enables the multiple MERGE operations between sourcedata stream 310 and the destination tables 320, 322, 324 to be carriedout using only the single scan of the source data structure.

Multi-Table Merge Operations with Source Augmentation

[0047]FIG. 4 illustrates an embodiment where a single scan of a sourcedata structure is used to yield source data stream for performingmultiple MERGE operations with different destination data structures,while enabling the source data stream to be augmented by one MERGEoperation before being fed into the subsequent MERGE operation.

[0048] In FIG. 4, the plan illustrates the manner in which source datastream 410 is merged into a first destination table 420 and then into asecond destination table 425. At time T0, a scan 412 has been performedof a source data structure that results in source data stream 410. Attime T0, a first destination scan 422 has also been completed of firstdestination table 420. Once the scans are complete, a first MERGEoperation 430 merges source data stream 410 into the first destinationtable 420. The first MERGE operation 430 uses the source data stream 410without modification to perform the first MERGE operation 430.

[0049] The source data stream 410 may introduce a new row into the firstdestination table 420 when the first MERGE operation 430 is performed.As an example, the new row may correspond to a new product. The firstMERGE operation 430 causes the source data stream 410 to receive newvalues that are to be provided in a column of the destination table. Thenew values may, for example, correspond to an identification number ofthe product. These new values may be generated from the firstdestination table 420. Then when the second MERGE operation 440 isperformed, the source data stream 410 includes the new values receivedfrom the first destination table 420. In this way, the source datastream 410 is augmented as a result of the first MERGE operation 430.

[0050] If first MERGE operation 430 is performed in the context of, forexample, a star schema (see description accompanying FIG. 5), thedestination table 420 will be altered by the MERGE operation, and thesource data stream 410 may be augmented for subsequent MERGE operations.For example, the first MERGE operation 430 may generate a new column forsource data stream 410 in response to the first MERGE operation beingperformed. This may occur when, for example, first destination table 420adds a set of dimension values to source data stream 410 prior toanother MERGE operation being performed on a different destination tablewhich uses the augmented source data stream.

[0051] In one embodiment, a result of performing first MERGE operation430 is that additional data is augmented to source data stream 410. Inan example provided by FIG. 4, the second MERGE operation 440 isperformed using the source data stream 410 after it is augmented as aresult of the first MERGE operation 430. The MERGE operation merges someor all of the augmented source data stream 410 into the seconddestination table 425. However, the source data stream 410 is modifiedby the first MERGE operation 430 without performing another scan of thesource data structure. Thus, the second MERGE operation 440 uses thescan 412 performed at time T0, and the second destination scan 428 ofthe second destination table 425 performed at time T1. No other scans ofthe source data structure is necessary other than the single scan 412performed at time T0 in order to perform the second MERGE operation 440.

[0052] The process by which source data stream 410 is augmented by firstMERGE operation 430 for second MERGE operation 440 may be repeated forsubsequent MERGE operations. A third set of instructions is illustratedbelow (in abbreviated form) for implementing such MERGE operations. 10*merge using <source> 20*  (merge using <source> 30*   when... 40*   else50*  producing <modified source>)

[0053] The third set of instructions illustrated above provide fornesting one merge command into another merge command so that, forexample, commands for executing the second MERGE operation 440 areexecuted using a return of the commands used to implement first MERGEoperation 430. The third set of instructions may incorporate commandsand concepts from other embodiments described herein. The result of thethird set of instruction is that a nested MERGE operation, provided bylines 20-50, returns a value for the second merge command, initiated online 10. This value corresponds to augmentations to the source datastream 410. The augmentation to the source data stream 410 is generatedon line 50, with the “producing” clause. The source data streamgenerated after the MERGE operation is identified by the “producing”clause columns. The “producing” clause columns can be either sourcestream columns or destinations columns of the corresponding MERGEoperation.

[0054] In database systems, for example, a star schema is distinguishedby the presence of one or more relatively large tables and severalrelatively smaller tables. Rather than duplicating the informationcontained in the smaller tables, the large tables contain references(foreign key values) to rows stored in the smaller tables. The largertables within a star schema are sometimes referred to as “fact tables”,while the smaller tables are sometimes referred to as “dimensiontables”. Typically, a series of MERGE operations merge the source datastream into a series of dimension tables, and finally into a fact table.Each dimension table augments one or more dimension values to the sourcestream to be used by the MERGE operations on the other dimension tablesand fact tables.

[0055]FIG. 5 illustrates a plan where an embodiment of the invention isimplemented in the context of a “star schema”. As with otherembodiments, at time T0, a scan 512 of the source data structure iscompleted in order to obtain the source data stream, and a scan 522 ofthe first dimension table 520. A first MERGE operation 540 is performedto augment dimensional data (e.g. columns) from the first dimensionaltable 520 to the source data stream. At time T1, a scan 524 of thesecond dimension table 525 may be performed. The source data stream 510with augmented data from the first dimension table 520 is merged intothe second dimension table 525 using a second MERGE operation 550.

[0056] Once the series of MERGE operations are performed to combine thedimension tables with the source data stream 510, the source data streamwith data augmented from the many dimension tables can be combined intofact table 530. At time T2, a scan 526 of the fact table 530 isperformed. The source data stream 510 containing data augmented fromprior operations with the dimension tables is then merged into the facttable 530.

[0057] Thus, FIG. 5 illustrates that an embodiment of the invention maybe implemented in a star schema, where a single scan of a source datastructure results in the source data stream that is then used forperforming a series of MERGE operations. With each MERGE operation, thesource data stream is augmented with data from one of the dimensiontables, until a final MERGE operation combines the source data streamwith the fact table.

Pipelined Merge Operations

[0058] Embodiments of the invention may be used to implement a“pipeline” in order to concurrently perform multiple MERGE operationsthat merge the source data stream into multiple destination tables. A“pipeline” refers to a mechanism where (i) all of a source data streamis subjected to each MERGE operation in a series of MERGE operations;(ii) sections of the source data stream are sequentially made availablewithout buffering the source stream to each MERGE operation, so thatwith the passage of time, each section has been subjected to all of theMERGE operations; and (iii) the source data stream (including all of theaugmentation) is pipelined through out all of the operations. In oneembodiment, another characteristic of a pipeline is that sections of thesource data stream are subjected to sequential MERGE operations in adesignated order. Thus, when a pipeline is implemented, at (i) aninitial time (T=0), the first section of the source data streamundergoes the first MERGE operation while no other section of the sourcedata stream is subjected to any such operation; and (ii) at a final time(T=final), the last section of the source data stream undergoes the lastMERGE operation while all other sections of the source data stream havealready undergone all of the MERGE operations. At any intermediate timeinterval between T=0 and T=final, the first section of the source datastream may undergo a MERGE operation that is further along in sequencethan the operation that the last section of the source data stream isbeing subjected to.

[0059]FIG. 6 illustrates a plan for providing a pipeline 680 forperforming multiple MERGE operations using a source data stream obtainedfrom a single scan of a source data structure. A first scan 612 resultsin the source data stream 610. Implementing the pipeline 680 results inthe source data stream 610 being structured into a sequential order thatis fed into a series of MERGE operations that are also sequenced. Thefollowing chronology may be used to explain the pipeline:

[0060] Before T=0, the scan 612 that results in the source data stream610 and the scan 622 of the first destination table 622 is completed.

[0061] At T=0, the first MERGE operation is initiated. A section of thesource data stream 610 undergoes the first MERGE operation 640 with thefirst destination table 620. The section of the source data stream 610that undergoes the first MERGE operation 640 is the first sequencedsection of the source data structure. The remainder of the source datastream 610 does not undergo the first MERGE operation 640.

[0062] Before T=1, the scan 624 of the second destination structure 625is completed.

[0063] At T=1, the first sequenced section of the source data stream 610undergoes the second MERGE operation 650 to combine its data with thesecond destination table 625. Simultaneously, a second sequenced sectionof the source data stream 610 undergoes the first MERGE operation 640 tocombine its data with the first destination table 620. The source datastream 610 other than the first and second sequenced sections do notundergo any MERGE operations.

[0064] Before T=2, a scan 626 of the third destination structure 630 iscompleted.

[0065] At T=2, the first sequenced section of the source data stream 610undergoes the third manipulation operation 66 to combine its data withthe third destination table 630. Simultaneously, (i) the secondsequenced section of the source data stream 610 undergoes the secondMERGE operation 650 to combine its data with the second destinationtable 625; and (ii) a third sequenced section of the source data streamundergoes the first MERGE operation 640 to combine its data with thefirst destination table 620. For purpose of explanation, it is assumedthat no other sections of the source data stream 610 remain.

[0066] At T=3 (not shown in the plan), the first sequenced section ofthe source data stream 610 has undergone all of the MERGE operations.The other sequenced sections of the source data stream 610 of iteratedto the next respective MERGE operation.

[0067] At T=4 (also not shown in the plan), the second sequenced sectionof the source data stream 610 has undergone all of the MERGE operations.The third sequenced section remains, and it is undergoing the thirdMERGE operation.

[0068] At T=5, all of the sequenced sections of the source data stream610 have undergone all of the MERGE operations.

[0069] In order to implement pipeline 680, the MERGE operations are (i)non-blocking, and (ii) preserve the source data stream. In order topreserve the source data stream 610, an outer-join may be performed.This type of MERGE operation is “non-blocking” for the source data 610because a particular section of the source table is not blocked fromfurther use in other MERGE operations once the first MERGE operation 630has been performed on that particular section.

[0070]FIG. 6 illustrates an embodiment where multiple MERGE operationsmay be performed concurrently, with only a single scan of the sourcedata structure. Such an embodiment greatly improves performance ofmultiple MERGE operations.

[0071] In order to implement pipelined merge operation, all theoperations, which are required to perform a MERGE operation at one nodeof the pipeline, should be non-blocking. Operations 640. 650, 660 in thecontext of FIG. 6 are non-blocking operations.

Lookup Node

[0072] A pipeline such as described above does not provide for alteringthe source data stream. But in certain applications like star-schemas, apipeline is beneficial, and data from destination tables (thedimensional tables in the star schema) need to be passed on for use withother MERGE operations. In such applications, a look-up node may beimplemented. The look-up node is a temporary data structure thatmaintains a set of data that is to augment the source data in subsequentMERGE operations.

[0073] The look-up node refers to a node that contains a temporary datastructure that stores data from a destination table, and can augment thesource data with the data contained in its data structure.

[0074]FIG. 7 illustrates a plan for implementing a look-up node, underan embodiment of the invention, the plan illustrates a first MERGEoperation 730 to combine data from a source data stream 710 with a firstdestination table 720. In order to perform the first MERGE operation730, a source table scan 712 is performed on a source data structure toyield the source data stream 710, and a first destination table scan 722is performed on the first destination table 720. The source table scan712 and the first destination table scan 722 may be completed by timeT0. The source table scan 712 is performed one time, and subsequentlyused for both the first MERGE operation 730 and the second MERGEoperation 740.

[0075] In an application such as a star schema, source data 710 isaugmented with modified data from each of the successive destinationtables. Thus, the second MERGE operation 740 receives augmented sourcedata 710, and the augmented source data is used for the second MERGEoperation 740. In order to perform the second MERGE operation 740, asecond destination table scan 727 is performed on the second destinationtable 725. But the source table scan 712 completed by time T0 is usedfor the source data 710 when performing the second MERGE operation 740.Thus, a single scan of source data 710 is used to perform multiple-MERGEoperations, even when source data 710 has been augmented.

[0076] In an embodiment, a first lookup node 750 provides a mechanism bywhich the source table scan 712 is preserved and augmented.Specifically, the first look-up node 750 stores data from the firstdestination table 720 that has been modified as a result of the firstMERGE operation 730. Once the first MERGE operation 750 is complete, thefirst look-up node 750 augments the modified data from the firstdestination table to the source data for use with the second MERGEoperation.

[0077] Likewise, the second look-up node 755 stores data from the seconddestination table 725 that has been modified as a result of the secondMERGE operation 740. The modified data in the second look-up node mayaugment the source data 710, which may already be augmented from thefirst look-up node 750. Thus, the third MERGE operation 750 is performedusing source data 710, augmented with modified data from the firstdestination table 720 and the second destination table 725.

[0078] According to one embodiment, lookup nodes 750, 755 are only usedwhen the plan for performing multiple MERGE operations calls foraugmenting the source data 710. Thus, the MERGE operations 730, 740 areconsidered as separate and independent operations from the lookup nodes750, 755 and the operations performed therein.

[0079]FIG. 8 illustrates a method for implementing a look-up node suchas described in FIG. 8 for use with the MERGE operation. A method suchas described in FIG. 8 is to be performed for a specific node, such asfirst look-up node 850.

[0080] In step 810, the outer-join operation of the first MERGEoperation 830 is completed. Step 815 makes a determination as to whethera source row that is to be used in the MERGE operation is to be anINSERT.

[0081] If the determination is negative, step 820 provides that thesource row is to be an UPDATE. Step 830 provides that old columns fromthe first destination table 820 are fetched. Step 840 provides that newcolumn values are computed for the row resulting from executing anUPDATE between the source row and the identified destination data.

[0082] Following a positive determination in step 815, or following step840, the result is that there is a new row for the first destinationtable 820. Step 845 provides that the new row is inserted into the firstlook-up node 850 hash table. Step 850 provides that the MERGE operationis performed as the first MERGE operation 830.

[0083] In step 860, the source row is augmented with columns from thefirst look-up node 850, which are stored in the hash table of that node.These columns represent changed values from the first destination table820.

[0084] Step 870 provides that the augmented source row is passed on tothe next MERGE operation. In FIG. 8, this may correspond to second MERGEoperation 840.

Hardware Overview

[0085]FIG. 9 is a block diagram that illustrates a computer system 9000upon which an embodiment of the invention may be implemented. Computersystem 9000 includes a bus 9002 or other communication mechanism forcommunicating information, and a processor 9004 coupled with bus 9002for processing information. Computer system 9000 also includes a mainmemory 9006, such as a random access memory (RAM) or other dynamicstorage device, coupled to bus 9002 for storing information andinstructions to be executed by processor 9004. Main memory 9006 also maybe used for storing temporary variables or other intermediateinformation during execution of instructions to be executed by processor9004. Computer system 9000 further includes a read only memory (ROM)9008 or other static storage device coupled to bus 9002 for storingstatic information and instructions for processor 9004. A storage device9010, such as a magnetic disk or optical disk, is provided and coupledto bus 9002 for storing information and instructions.

[0086] Computer system 9000 may be coupled via bus 9002 to a display9012, such as a cathode ray tube (CRT), for displaying information to acomputer user. An input device 9014, including alphanumeric and otherkeys, is coupled to bus 9002 for communicating information and commandselections to processor 9004. Another type of user input device iscursor control 9016, such as a mouse, a trackball, or cursor directionkeys for communicating direction information and command selections toprocessor 9004 and for controlling cursor movement on display 9012. Thisinput device typically has two degrees of freedom in two axes, a firstaxis (e.g., x) and a second axis (e.g., y), that allows the device tospecify positions in a plane.

[0087] The invention is related to the use of computer system 9000 forimplementing the techniques described herein. According to oneembodiment of the invention, those techniques are performed by computersystem 9000 in response to processor 9004 executing one or moresequences of one or more instructions contained in main memory 9006.Such instructions may be read into main memory 9006 from anothercomputer-readable medium, such as storage device 9010. Execution of thesequences of instructions contained in main memory 9006 causes processor9004 to perform the process steps described herein. In alternativeembodiments, hard-wired circuitry may be used in place of or incombination with software instructions to implement the invention. Thus,embodiments of the invention are not limited to any specific combinationof hardware circuitry and software.

[0088] The term “computer-readable medium” as used herein refers to anymedium that participates in providing instructions to processor 9004 forexecution. Such a medium may take many forms, including but not limitedto, non-volatile media, volatile media, and transmission media.Non-volatile media includes, for example, optical or magnetic disks,such as storage device 9010. Volatile media includes dynamic memory,such as main memory 9006. Transmission media includes coaxial cables,copper wire and fiber optics, including the wires that comprise bus9002. Transmission media can also take the form of acoustic or lightwaves, such as those generated during radio-wave and infra-red datacommunications.

[0089] Common forms of computer-readable media include, for example, afloppy disk, a flexible disk, hard disk, magnetic tape, or any othermagnetic medium, a CD-ROM, any other optical medium, punchcards,papertape, any other physical medium with patterns of holes, a RAM, aPROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, acarrier wave as described hereinafter, or any other medium from which acomputer can read.

[0090] Various forms of computer readable media may be involved incarrying one or more sequences of one or more instructions to processor9004 for execution. For example, the instructions may initially becarried on a magnetic disk of a remote computer. The remote computer canload the instructions into its dynamic memory and send the instructionsover a telephone line using a modem. A modem local to computer system9000 can receive the data on the telephone line and use an infra-redtransmitter to convert the data to an infra-red signal. An infra-reddetector can receive the data carried in the infra-red signal andappropriate circuitry can place the data on bus 9002. Bus 9002 carriesthe data to main memory 9006, from which processor 9004 retrieves andexecutes the instructions. The instructions received by main memory 9006may optionally be stored on storage device 9010 either before or afterexecution by processor 9004.

[0091] Computer system 9000 also includes a communication interface 9018coupled to bus 9002. Communication interface 9018 provides a two-waydata communication coupling to a network link 9020 that is connected toa local network 9022. For example, communication interface 9018 may bean integrated services digital network (ISDN) card or a modem to providea data communication connection to a corresponding type of telephoneline. As another example, communication interface 9018 may be a localarea network (LAN) card to provide a data communication connection to acompatible LAN. Wireless links may also be implemented. In any suchimplementation, communication interface 9018 sends and receiveselectrical, electromagnetic or optical signals that carry digital datastreams representing various types of information.

[0092] Network link 9020 typically provides data communication throughone or more networks to other data devices. For example, network link9020 may provide a connection through local network 9022 to a hostcomputer 9024 or to data equipment operated by an Internet ServiceProvider (ISP) 9026. ISP 9026 in turn provides data communicationservices through the world wide packet data communication network nowcommonly referred to as the “Internet” 9028. Local network 9022 andInternet 9028 both use electrical, electromagnetic or optical signalsthat carry digital data streams. The signals through the variousnetworks and the signals on network link 9020 and through communicationinterface 9018, which carry the digital data to and from computer system9000, are exemplary forms of carrier waves transporting the information.

[0093] Computer system 9000 can send messages and receive data,including program code, through the network(s), network link 9020 andcommunication interface 9018. In the Internet example, a server 9030might transmit a requested code for an application program throughInternet 9028, ISP 9026, local network 9022 and communication interface9018.

[0094] The received code may be executed by processor 9004 as it isreceived, and/or stored in storage device 9010, or other non-volatilestorage for later execution. In this manner, computer system 9000 mayobtain application code in the form of a carrier wave.

[0095] In the foregoing specification, the invention has been describedwith reference to specific embodiments thereof. It will, however, beevident that various modifications and changes may be made theretowithout departing from the broader spirit and scope of the invention.The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense.

What is claimed is:
 1. A method for combining data within a databasesystem, the method comprising: performing a first merge operation tomerge source data from a source data structure with a first destinationdata structure, wherein the source data structure and the destinationdata structure are relational data structures; and initiating a secondmerge operation to merge a set of data that is returned by the firstmerge operation as a portion of the source data with a seconddestination data structure, wherein the second merge operation isinitiated independent of any other portion of the source data undergoingthe first merge operation.
 2. The method of claim 1, wherein initiatinga second merge operation includes performing the second merge operationon the portion of the source data while the first merge portion isongoing with a remainder of the source data.
 3. The method of claim 1,further comprising the step of performing a scan of the source datastructure in order to obtain the source data, and wherein the steps ofperforming a first merge operation and initiating a second mergeoperation are performed using only the scan of the source data structurewithout performing any other scan of the source data structure.
 4. Themethod of claim 1, wherein the step of initiating a second mergeoperation is performed prior to completion of the first merge operationto merge all of the source data from the source data structure with thefirst destination data structure.
 5. The method of claim 1, wherein thesource data structure is a table.
 6. The method of claim 1, whereininitiating a second merge operation to merge a set of data that isreturned by the first merge operation as a portion of the source dataincludes initiating the second merge operation using the portion of thesource data, wherein the portion of the source data structure isunmodified by the first merge operation.
 7. The method of claim 1,wherein initiating a second merge operation to merge the set of dataincludes initiating the second merge operation by streaming the sourcedata from the first merge operation to the second merge operationwithout ever storing the source data.
 8. The method of claim 2, whereinperforming a first merge operation includes identifying whether anydifferences exist between the portion of the source data and the firstdestination data structure, and modifying the first destination datastructure based on any identified differences.
 9. The method of claim 8,wherein modifying the first destination data structure includes updatingthe first destination data structure using the portion of the sourcedata.
 10. The method of claim 8, wherein modifying the first destinationdata structure includes inserting the portion of the source data intothe first destination data structure.
 11. The method of claim 10,wherein modifying the first destination data structure includes deletingat least part of the data in the destination data structure based ondata from the portion of the source data being merged with the firstdestination data structure.
 12. The method of claim 7, wherein streamingthe source data from the first merge operation to the second mergeoperation is performed at a constant rate.
 13. The method of claim 1,further comprising augmenting the source data as a result of the firstmerge operation and then performing the second merge operation using theaugmented source data.
 14. A method for combining data within a databasesystem, the method comprising: implementing a pipeline for mergingsource data with a series of destination data structures; and using thepipeline to perform a merge operation with each of the destination datastructures in the series.
 15. The method of claim 14, further comprisingaugmenting the source data as a result of performing the merge operationwith one or more of the destination data structures.
 16. The method ofclaim 14, wherein performing the merge operation with each of thedestination data structures includes identifying whether any differencesexist between the portion of the source data and each of the destinationdata structure, and modifying that destination data structure based onany identified differences
 17. The method of claim 14, whereinperforming the merge operation with each of the destination datastructures includes performing one of a MERGE, UPDATE, or INSERToperation.
 18. A computer-readable medium carrying one or more sequencesof instructions for combining data within a database system, whereinexecution of the one or more sequences of instructions by one or moreprocessors causes the one or more processors to perform the steps of:performing a first merge operation to merge source data from a sourcedata structure with a first destination data structure, wherein thesource data structure and the destination data structure are relationaldata structures; and initiating a second merge operation to merge a setof data that is returned by the first merge operation as a portion ofthe source data with a second destination data structure, wherein thesecond merge operation is initiated independent of any other portion ofthe source data undergoing the first merge operation.
 19. Thecomputer-readable medium of claim 18, wherein the step of initiating asecond merge operation includes the step of performing the second mergeoperation on the portion of the source data while the first mergeportion is ongoing with a remainder of the source data.
 20. Thecomputer-readable medium of claim 18, further comprising the step ofperforming a scan of the source data structure in order to obtain thesource data, and wherein the steps of performing a first merge operationand initiating a second merge operation are performed using only thescan of the source data structure without performing any other scan ofthe source data structure.
 21. The computer-readable medium of claim 18,wherein the step of initiating a second merge operation is performedprior to completion of the first merge operation to merge all of thesource data from the source data structure with the first destinationdata structure.
 22. The computer-readable medium of claim 18, whereinthe source data structure is a table.
 23. The computer-readable mediumof claim 18, wherein the step of initiating a second merge operation tomerge a set of data that is returned by the first merge operation as aportion of the source data includes initiating the second mergeoperation using the portion of the source data, wherein the portion ofthe source data structure is unmodified by the first merge operation.24. The computer-readable medium of claim 18, wherein the step ofinitiating a second merge operation to merge the set of data includesinitiating the second merge operation by streaming the source data fromthe first merge operation to the second merge operation without everstoring the source data.
 25. The computer-readable medium of claim 19,wherein the step of performing a first merge operation includesidentifying whether any differences exist between the portion of thesource data and the first destination data structure, and modifying thefirst destination data structure based on any identified differences.26. The computer-readable medium of claim 25, wherein the step ofmodifying the first destination data structure includes updating thefirst destination data structure using the portion of the source data.27. The computer-readable medium of claim 25, wherein the step ofmodifying the first destination data structure includes inserting theportion of the source data into the first destination data structure.28. The computer-readable medium of claim 27, wherein the step ofmodifying the first destination data structure includes deleting atleast part of the data in the destination data structure based on datafrom the portion of the source data being merged with the firstdestination data structure.
 29. The computer-readable medium of claim24, wherein the step of streaming the source data from the first mergeoperation to the second merge operation is performed at a constant rate.30. The computer-readable medium of claim 18, further comprising thestep of augmenting the source data as a result of the first mergeoperation and then performing the second merge operation using theaugmented source data.